The Computer Chess Reports (CCR) 1 hour test is a set of 25 chess problems designed to give a quick estimate of a computer program's strength. I used the set published in CCR v5n1, pp. 25-29, and used the scoring formula (estimated USCF rating = 1880 + 10 times test score) given in CCR v5n2, p 43.
To test MacChess 2.0, I left the memory allocation of MacChess 2.0 at 2Mb, and ran it on a Macintosh 6100/66 with Speed Doubler on, all other extensions off. The 6100/66 is a 66Mhz machine with secondary cache. Since MacChess 2.0 is written in 680x0 code, Speed Doubler should have a significant influence. For comparison, I also ran problems 23 and 24 with Speed Doubler off (2.7x slower) and on a Quadra 610 (25Mhz 68040 with cache card) (1.75x slower than 6100/66 with Speed Doubler).
Notes on effect of machine speed
The reader may wonder how strong MacChess 2.0 is on their machine, but not curious enough to run the full test as described below. Since MacChess 2.0 has been benchmarked on this test, and the relationship between speed and strength is fairly well established, I suggest that those so inclined simply follow the instructions below for problem 23 only, and note the time used by the program to find the correct move (Bd3xh7). By comparing this time to that used by the 6100/66 with Speed Doubler, we can scale to estimate the strength of MacChess 2.0 in a particular configuration. (Note: for problems 23 and 24, the ratios of times were consistent both between different configurations and within a configuration for the three configurations tested.)
The basic test yielded an estimated USCF rating of 1880 + 10 x 43 = 2310. Note that this test is far from rigorous, but has been tabulated for many programs, and on the whole, correlates well with ratings obtained from actual play.
The test instructions:
- set the level to infinite (use the "Level" menu, "Set Level" option, select "Infinite")
- turn off the opening book ( I moved the file "MacChess Openings book" outside the "MacChess" folder temporarily. I couldn't figure out how to do this for Chessmaster 3000, so I didn't do a direct comparison)
- turn off thinking on the opponent's time (use the "Preferences" menu, second item)
- use player-player mode to set up the position (use the "File" menu, "Open Game" option to load the problem)
- turn on the display or hint feature (I used the "Principal Variation" (PV) window)
- begin the search
- Note the move choice at 15 seconds, 30 seconds, 1 minute and 2 minutes.
- Score 1 point for each correct move found or bad move avoided at each of the four time
intervals. If the move is found at 15", not at 30" or 1', and found again at 2', the program
would get 2 points.
- To get an estimated USCF rating, multiply the test score by 10 and add to 1880.
[ revised fit from CCR V5N2]
(note: British ratings = USCF - 100, Swedish = USCF - 180)
The Answers (and points scored by MacChess 1.0 and 2.0)
{problem number, points scored by MacChess 1.0, points scored by MacChess 2.0, solution}
1] 0 0 Qb3
2] 0 1 Bc8
3] 0 3 Nh6
4] 0 0 b4
5] 2 3 e5
6] 4 2 Bxc3+
7] 0 1 Rfb8
8] 0 0 d5
9] 0 1 Nd4
10] 4 4 a4
11] 0 0 d5
12] 4 4 bxf7+
13] 4 4 avoid Nxe4
14] 3 4 avoid Nxc4
15] 1 1 exf6
16] 0 0 d5
17] 1 0 avoid hxg4
18] 0 1 Bxf6
19] 0 0 Bxe6
20] 0 3 Nd4-b5
21] 0 4 avoid Kxe6
22] 0 0 a4
23] 0 2 Bxh7+
24] 1 1 Nxe5
25] 4 4 avoid Qxb3
--------------------
28 43 Total points for MacChess 1.0, 2.0
Comments on the difference between MacChess 1.0 and 2.0
The improvement between versions 1.0 and 2.0 were impressive - from 2160 to 2310. The author's comments indicated that the tactical game was improved. Definitely! Note that on a couple of problems, #6 and #17, version 2.0 actually does worse than 1.0. On problem 6, version 2.0 switches from preferring the best line to an inferior line, before going back. This actually happened to version 1.0 as well, but since version 1.0 took more than 2 minutes to reach the point of preferring the inferior line, it isn't reflected in the test score.
The author indicates plans to improve the program's positional and endgame play. This test set did try to evaluate positional play, but not endgame play. CCR does show a set to test endgame strength, but it takes over 6 hours to run, so I'll let someone else have the honor.
About the reviewer
Richard Fowell retired from tournament chess after 13 years with a USCF rating of 2210 (peak rating: 2325). He has used Chessmaster 2000, 2100 and 3000 on the Mac, as well as Sargon IV, various versions of GNUChess, and MacChess. He is a faithful (5 yr) subscriber to Computer Chess Reports (CCR), which he recommends to those interested in computer chess. The rating estimates above are taken from CCR. Richard can be contacted on the Internet at fowell@netcom.com.
Computer Chess Reports
This publication is edited by International Master Larry Kaufman. Subscriptions are available through the computer chess retailer ICD at $18.00 for four issues (ca. 40 pp/issue). Phone number is 1-800-645-4710 (516-424-3300 outside U.S.). ICD carries lots of chess machines and DOS/Windows chess software.
